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EXECUTIVE SUMMARY 


There is growing interest in leveraging Title IV student aid to improve college completion. Advocates have proposed linking 
funding in the Pell and Campus-Based Programs to measures of college performance. However, to do so in an equitable and 
efficient manner, raw measures of college output, such as rates of graduation and academic progress, must be adjusted to 
reflect differences in factors that determine those rates - inputs such as college mission, student characteristics, and other 
constraints. Failing to account for inputs when measuring and evaluating college performance unfairly penalizes colleges that 
are efficiently serving large numbers of low-income students, particularly colleges doing so with limited resources. 


To demonstrate the impact that serving low-income students can have on raw measures of college performance, this bulletin 
explores the relationship between 6-year graduation rate and three inputs: the percentage of first-time students who are Pell 
recipients, average test score of the student body, and level of endowment per student. The analysis finds that these three 
inputs are powerful determinants of 6-year graduation rates at nonprofit 4-year public and private colleges. 


Empirical 

Findings 


As the percentage of a college’s students who are Pell recipients (serving Pell recipients) rises, 6-year 
graduation rate declines from 80% to 25%, and average test score declines from 29 to 19. (Figure 1) 

Serving Pell recipients and average test score, combined, account for 76% of the observed variation in 
6-year graduation rates of 4-year public and private colleges. (Table 1) 

As endowment per student falls, serving Pell recipients lowers average 6-year graduation rate from 
67% to 28% and from 85% to 33% at public and private colleges, respectively. (Tables 2-A and B) 

Serving very high percentages of Pell recipients, with very low endowment per student, reduces 
average 6-year graduation rate to 23% at public colleges and 19% at private colleges. (Table 2-C) 


The more a 4-year college defines its mission as serving low-income students, and the more modest its endowment per 
student, the more its 6-year graduation rate will deviate from a valid and reliable assessment of its relative performance. 


To illustrate the importance of adjusting raw 6-year graduation rate for these effects, the analysis uses one particular model 
and dataset to calculate an estimate of value added for ten selected colleges. Reordering the colleges by this estimate reverses 
their ranking by raw 6-year graduation rate, with those serving Pell recipients perceived as most effective. (Table 3) 


However, estimates of value added can vary considerably, depending on the statistical model and dataset used to generate the 
estimates. Most important, no model will explain all of the variation in 6-year graduation rates and no dataset will contain 
observations on all of the important student characteristics or resource constraints that determine those rates. 


These findings have major implications for student aid policy, including reauthorization of the Higher Education Act (HEA). 


• Using raw output measures, such as rates of graduation or student academic progress, in the awarding 
or allocation of Title IV student aid will harm low-income students and the colleges that serve them. 


Policy 

Implications 


• To prevent such harm, output measures must be adjusted to adequately reflect differences in inputs, in 
particular, college mission, student characteristics, resources, and factors beyond colleges’ control. 

• Because estimates of value added are vulnerable to both modeling and data limitations, such measures 
should not be used to support high-stakes decisions in either the Pell or Campus-Based Programs. 


• The best approach to improve graduation rates is to use well-designed case studies to identify policies 
and practices shown to be effective at peer colleges and incentivize colleges to implement them. 


In race-to-the-top competitions for additional funds, colleges should be required to compete only against peers with highly 
comparable inputs. These inputs include the percentage of students who are Pell recipients, average test score, endowment 
per student, and other factors shown to have a statistically significant relationship to the output measure in question. 


The empirical findings and policy implications above are not meant to imply that 4-year colleges serving a large number of 
Pell recipients should not be evaluated or that graduation rates cannot or should not be improved. All colleges must meet 
minimum standards of performance, implement policies and practices shown to be effective in improving graduation rates at 
peer colleges, and provide adequate consumer protections, including accurate information for students and parents. 
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THE IMPACT ON GRADUATION RATES 
OF SERVING PELL GRANT RECIPIENTS 


Graduation rates are highly imperfect indicators of college performance. In their raw, unadjusted form, the rates can fail to 
measure quality and content of curriculum, level of student preparedness or employability, or the extent and value of student 
learning. In addition. 


Limitations of 
Graduation Rates 


• Those rates which are readily available include only first-time, full-time, degree-seeking students and 
fail to account adequately for transfer students or part-time students. 

• Because of their restricted time frame, rates can exclude long-term degree and certificate attainment 
and fail to reflect current college performance. 


• Finally, using the rates to award and distribute program funding can create perverse incentives to limit 
enrollment of at-risk students or lower standards of graduation. 


For more information on graduation rate as a measure of college performance, see American Association of State Colleges 
and Universities (2002), Bailey and Xu (2012), Gold and Albert (2006), Kelchen and Harris (2012), and McGuire (2012). 

However, in spite of these weaknesses, advocates have called for using graduation rates in the awarding and distribution of 
Title IV student aid. For example, to improve college completion, proposals have called for adjusting the level of Pell awards 
or reallocating some portion of Campus-Based Funding among colleges on the basis of graduation rates. However, thus far, 
no proposal has shown if and how the rates must be adjusted to avoid harm to colleges serving low-income students. 

Issues and Questions 

Because the issue of using graduation rates in the awarding and distribution of need-based grant aid may be considered in the 
upcoming HEA reauthorization, it is essential to examine the determinants of those rates, how they must be adjusted to fairly 
and accurately indicate relative college performance, and what impact those adjustments are likely to have on perceptions of 
college effectiveness and distribution of Title IV funds. 

This bulletin addresses the following specific issues and questions. 


Issues and 
Questions 


• Overall Impact of Serving Pell Grant Recipients. To what extent does a 4-year college’s graduation 
rate depend upon the number of Pell recipients it serves? For example, to what extent does the percent 
of first-time students who are Pell recipients determine test score and 6-year graduation rate? 

• Factors Determining 6-Year Graduation Rates. Which factors explain the 6-year graduation rate of 
4-year colleges in a multivariate analysis? Which factors appear to be most important, and how much 
variation do they explain? Which factors appear to be relatively unimportant? 

• Impact of Percent Pell Recipients by Endowment per Student. To what extent does the relationship 
between percent of first-time students who are Pell recipients, test score, and 6-year graduation rate 
vary by endowment per student at public and private 4-year colleges? 

• Colleges with Most Pell Recipients and Least Resources. Among 4-year colleges with the lowest 
endowment per student and highest percentage of Pell recipients, what is the relationship between test 
score and 6-year graduation rate? To what extent does test score reduce the 6-year graduation rate? 

• Actual Rates Minus Predicted Rates = Value Added. If a multivariate statistical model is used to 
predict the 6-year graduation rate of each college, and the predicted rate is subtracted from the actual 
rate to yield value added, to what extent are perceptions of college performance and ranking altered? 


These issues and questions are addressed using data from the Integrated Postsecondary Education Data System (IPEDS), 
cross-tabular analyses, and a step-up regression model aimed at identifying the determinants of 6-year graduation rates. 
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Percent of Students 
Who Are Pell Recipients 
Is a Strong Determinant of 
Average Test Score and 
6- Year Graduation Rate 


FIGURE 1: SIX-YEAR GRADUATION RATE AND 
TEST SCORE BY PERCENT PELL RECIPIENTS 
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OVERALL IMPACT OL SERVING PELL GRANT RECIPIENTS 

Using IPEDS data from 2010, Figure 1 shows the inverse relationship between the percentage of first-time students who are Pell 
recipients, average test score, and 6-year graduation rate at nonprofit 4-year public and private colleges. 

• On the far left, 4-year colleges with up to 20% Pell recipients have an average 6-year graduation rate of 80% and an average 
test score of 29. 

• On the far right, 4-year colleges with 80% or more Pell recipients have an average 6-year graduation rate of 25% and an 
average test score of 19. 

The more a 4-year college serves Pell recipients, the lower its 6-year graduation rate and the more difficult it is to improve that rate. 


TABLE 1: DETERMINANTS OF 6- YEAR GRADUATION RATES AT 4- YEAR COLLEGES 


[ Inputs 

Model 1 

Model 2 

Model 3 

Model 4 

Model 5 

Model 6 

Model 7 

Model 8 

Percent Pell Recipients 

-0.774*** 

0.145*** 

-0.162*** 

-0.178*** 

-0.205*** 

-0.203*** 

-0.227*** 

-0.225*** 

ACT 75 th Percentile Composite 


4.171*** 

3.682*** 

3.097*** 

2.900*** 

2.927*** 

2.751*** 

2.745*** 

Percent Part-Time 



-0.263*** 

-0.249*** 

-0.231*** 

-0.230*** 

-0.240*** 

-0.241*** 

4-Year College Endowment (In) 




1.507*** 

1.586*** 

1.223*** 

1.330*** 

1.336*** 

4- Year College Control 





3.003*** 

4.472*** 

5.294*** 

5.289*** 

Full-Time Enrollment 






0.021*** 

0.021*** 

0.023*** 

Expenditures Per Student 







0.003 

0.003 

Percent Minority 








-0.003 

R 2 

0.551 

0.761 

0.791 

0.798 

0.804 

0.806 

0.808 

0.808 

Adjusted R 2 

0.551 

0.761 

0.790 

0.797 

0.803 

0.805 

0.807 

0.807 

Observations 

1,334 

1,334 

1,331 

1,277 

1,277 

1,277 

1,231 

1,231 


Source: Calculated from 2010 data in the Integrated Postsecondary Education Data System (IPEDS). 

*p < 0:05. **p < 0:01. ***p < 0:001 

FACTORS DETERMINING 6- YEAR GRADUATION RATES 


Percent of Students 
Who Are Pell Recipients 
and Test Score Account 
for Over 76% of the 
Variation in 6- Year 
Graduation Rates 


The results of a step-up regression of 6-year graduation rate on various inputs are shown in Table 1. 

• As might be expected from Figure 1, the percent of students who are Pell recipients explains 
over half of the variance in 6-year graduation rate: (R-Squared = 0.551). 

• When test score is added, over 75% of the variation is explained: (R-Squared = 0.761). 

• Adding the variables of percent part-time, endowment, control (public/private), and full-time 
enrollment increases R-Squared modestly by 0.045 to 0.806. 

Expenditures per student and percent minority are not statistically significant. 
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TABLE 2-A: IMPACT ON GRADUATION RATES OF SERVING PELL RECIPIENTS 

By Endowment Per Student 
4- Year Public Colleges 


Endowment Per Student (FTE) 


Percent of 
Students 
Who Are Pell 

Highest Quartile 
(M = $34,000) 

Third Quartile 
(M = $6,000) 

Second Quartile 
(M = $3,000) 

Lowest Quartile 
(M = $1,000) 

Recipients 

Average 
6-Yr. Rate 

Test 

Score 

Average 
6-Yr. Rate 

Test 

Score 

Average 
6-Yr. Rate 

Test 

Score 

Average 
6-Yr. Rate 

Test 

Score 

Under 30% 

67% 

28 

61% 

26 

60% 

26 

54% 

24 

30% to 39% 

51% 

26 

49% 

25 

44% 

24 

43% 

23 

40% to 49% 

47% 

24 

37% 

23 

36% 

22 

41% 

21 

50% and over 

31% 

21 

30% 

20 

29% 

20 

28% 

20 


Source: Calculated from 2010 data in the Integrated Postsecondary Education Data System (IPEDS). 


The Impact of 
Serving Pell Recipients 
on 6- Year Graduation Rate 
Intensifies as Endowment 
per Student Declines 
at Public Colleges 


IMPACT OF PERCENT PELL RECIPIENTS BY ENDOWMENT PER STUDENT: 4-YEAR PUBLIC COLLEGES 

Table 2-A shows the impact of serving Pell recipients on average 6 -year graduation rate and average test score of 4-year public 
colleges, by endowment per student in quartiles. 

• For colleges in the highest quartile of endowment per student, as the percent of students who are Pell recipients rises, average 
6-year graduation rate falls from 67% to 31% and average test score falls from 28 to 21. 

• For those in the lowest quartile, 6-year graduation rate falls from 54% to 28% and average test score falls from 24 to 20. 

Overall, as the percent of students who are Pell recipients rises and endowment per student falls, average 6-year graduation rate falls 
from 67% to 28% and average test score falls from 28 to 20. 


TABLE 2-B: IMPACT ON GRADUATION RATES OF SERVING PELL RECIPIENTS 

By Endowment Per Student 
4- Year Private Colleges 


The Impact of 
Serving Pell Recipients 
on 6- Year Graduation Rate 
Intensifies as Endowment 
per Student Declines 
at Private Colleges 


Source: Calculated from 2010 data in the Integrated Postsecondary Education Data System (IPEDS). 
*Note: Ranges differ slightly from those used for public colleges above. 



j Endowment Per Student (FTE) | 

Percent of 
Students 
Who Are Pell 

Highest Quartile 
(M = $269,000) 

Third Quartile 
(M = $35,000) 

Second Quartile 
(M = $16,000) 

Lowest Quartile 
(M = $5,000) 

Recipients* * 

Average 
6-Yr. Rate 

Test 

Score 

Average 
6-Yr. Rate 

Test 

Score 

Average 
6-Yr. Rate 

Test 

Score 

Average 
6-Yr. Rate 

Test 

Score 

Under 20% 

85% 

31 

77% 

28 

68% 

27 

58% 

26 

20% to 34% 

70% 

28 

64% 

27 

58% 

26 

52% 

25 

35% to 49% 

58% 

26 

51% 

25 

50% 

24 

48% 

23 

50% and over 

44% 

23 

41% 

22 

38% 

21 

33% 

20 


IMPACT OF PERCENT PELL RECIPIENTS BY ENDOWMENT PER STUDENT: 4- YEAR PRIVATE COLLEGES 

Table 2-B shows the impact of serving Pell recipients on average 6-year graduation rate and average test score of 4-year private 
colleges, by endowment per student in quartiles. 

• For colleges in the highest quartile of endowment per student, as the percent of students who are Pell recipients rises, average 
6-year graduation rate falls from 85% to 44% and average test score falls from 31 to 23. 

• For those in the lowest quartile, 6-year graduation rate falls from 58% to 33% and average test score falls from 26 to 20. 

Overall, as the percent of students who are Pell recipients rises and endowment per student falls, average 6-year graduation rate falls 
from 85% to 33% and average test score falls from 31 to 20. 
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TABLE 2-C: IMPACT ON GRADUATION RATES OF SERVING FELL RECIPIENTS 


4- Year Colleges with Highest Percentage of Pell 
Recipients and Lowest Endowment Per Student 


At Colleges with 
Highest Percentage of 
Pell Recipients and Lowest 


Average 6-Year Graduation Rate if Test Score Is: 

4- Year College Control 

23 and above 

21-22 

19-20 

18 and below 

Endowment Per Student, 
Test Score Drives Graduation 

Public 

34% 

32% 

27% 

23% 

Rate to Lowest Levels 

Private 

44% 

39% 

30% 

19% 


Source: Calculated from 2010 data in the Integrated Postsecondary Education Data System (IPEDS). 

COLLEGES WITH MOST PELL RECIPIENTS AND LEAST RESOURCES 


Table 2-C shows the relationship between average test score and average 6-year graduation rate at 4-year public and private colleges 
with lowest endowment per student and highest percentage of students who are Pell recipients - 50% and over. 

• At 4-year public colleges with the lowest endowment per student and highest percentage of students who are Pell recipients, 
as test score decreases from 23 and above to 18 and below, average 6-year graduation rate falls from 34% to 23% . 

• At 4-year private colleges with the lowest endowment per student and highest percentage of students who are Pell recipients, 
as test score decreases from 23 and above to 18 and below, average 6-year graduation rate falls from 44% to 19% . 

The more a 4-year college - public or private - defines its mission as serving Pell Grant recipients, and the lower its endowment per 
student, the lower its 6-year graduation rate can be driven by declining average test score. 


TABLE 3: ADJUSTING 6- YEAR GRADUATION RATES FOR INPUTS 
Selected 4- Year Colleges 


College 

Actual 
6- Year Rate 




Inputs 



Predicted 
6- Year Rate 

Value 

% Pell 
Recipients 

Test 

Score 

Control 

Part-Time 

Endowment 

(In) 

% 

Minority 

Added 

A 

87 

6 

35 

0 

Private 

21 

8 

99 

-12 

B 

70 

11 

31 

23 

Private 

21 

9 

81 

-11 

C 

68 

20 

29 

6 

Public 

21 

7 

75 

-7 

D 

61 

23 

26 

10 

Public 

20 

19 

63 

-2 

E 

60 

34 

27 

15 

Private 

17 

5 

61 

-1 

F 

58 

39 

25 

12 

Public 

17 

30 

52 

+6 

G 

56 

41 

24 

12 

Public 

15 

23 

45 

+11 

H 

49 

47 

20 

21 

Private 

13 

16 

32 

+17 

I 

42 

72 

20 

37 

Public 

17 

95 

23 

+19 

J 

41 

76 

19 

13 

Public 

14 

94 

21 

+20 


Source: Calculated from 2010 data in the Integrated Postsecondary Education Data System (IPEDS). 


ACTUAL RATES MINUS PREDICTED RATES = VALUE ADDED 

Table 3 uses IPEDS data and the regression equation in Table 1 to illustrate how adjusting actual 6-year graduation rate for inputs 
yields a predicted 6-year rate, measure of value added, and new ranking. Note that actual 6-year rate (second column) minus the 
predicted rate (second-last column) equals value added (last column). 

• Ordering the ten colleges by value added, rather than by actual 6 -year graduation rate, completely reverses their ranking. 

• Colleges serving high proportions of Pell Grant recipients show the highest value added and ranking after adjustment. 

Table 3 is an illustration using one model and one data set. Alternative models and data will yield different results. Also, these ten 
colleges have been selected to show the potential impact of adjusting outputs for inputs. Not all colleges serving large numbers of Pell 
recipients have positive value added, nor do all colleges serving few Pell recipients have negative value added. 

Regardless of the model used, redistributing program funds on the basis of value added will shift funds away from colleges with high 
6-year graduation rates (but negative value added) toward colleges with low 6-year graduation rates (but positive value added). 
Accordingly, redistributing need-based grant aid on the basis of value added could have unexpected and unpopular consequences. 
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POLICY IMPLICATIONS AND LIMITATIONS 


These finding have major implications for student aid policy, including HEA reauthorization. 


• Using raw output measures, such as rates of graduation or student academic progress, in the 
awarding or allocation of Title IV student aid will harm low-income students and the colleges 
that serve them. 


Policy 

Implications 


• To prevent such harm, output measures must be adjusted to adequately reflect differences in 
inputs, in particular, college mission, student characteristics, resources, and factors beyond 
colleges’ control. 

• Because estimates of value added are vulnerable to both modeling and data limitations, such 
measures should not be used to support high-stakes decisions in either the Pell or Campus- 
Based Programs. 


• The best approach to improve graduation rates at 4-year public and private colleges is to use 
well-designed case studies to identify policies and practices shown to be effective at peer 
colleges and incentivize colleges to implement them. 

In race-to-the-top competitions for additional funds, colleges should be required to compete only against peers with 
highly comparable inputs. These inputs include the percentage of students who are Pell recipients, average test 
score, endowment per student, and other factors shown to have a statistically significant relationship to the output 
measure in question. 

Limitations 


The approach, model, and data used in this analysis are not meant to be definitive but, rather, illustrative of the need 
to adjust 4-year college outputs, such as graduation rates and measures of academic progress, for inputs. In general, 
techniques that adjust output for inputs, create measures of value added, and define peer groups within which to 
compare colleges are only as valid and reliable as the data used to do so. Regardless of the model used, differences 
in performance among colleges that appear to be similar might be attributable to differences in unobserved student 
characteristics or resource constraints. These differences cannot be identified centrally by multivariate analysis. 

To ensure that adjusted outputs, estimates of value added, and definitions of peer group are sufficiently precise for 
use in the awarding and allocation of Title IV student aid, far more research is needed. 

• Regression models used to determine inputs and adjust outputs must be improved. 

• Measures of college outcomes and inputs must be better defined and measured. 

• Alternative definitions of value added must be explored and evaluated. 

• Changes in the level of input-adjusted output, by college, must be developed. 

• Data required to more accurately measure outputs and inputs must be collected. 

The Advisory Committee encourages and welcomes further analysis of the impact of serving Pell Grant recipients 
on college graduation rates. 

The empirical findings and policy implications in this bulletin are not meant to imply that 4-year colleges serving a 
large number of Pell recipients should not be evaluated or that graduation rates cannot or should not be improved. 
All colleges must meet minimum standards of performance, implement policies and practices shown to be effective 
in improving graduation rates at peer colleges, and provide adequate consumer protections, including accurate 
information to students and parents. 


Five 

Areas for 
Further 
Research 
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